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VIDEO SURVEILLANCE SYSTEM AND METHOD 



TECHNICAL FIELD QF THE INVENTION 

This invention relates in general to surveillance 
and communication systems, and more specifically to a 
video surveillance system and method. 

5 

BACKGROUND OF THE INVENTION 

A point-of-sale (POS) device, an automated teller 
machine (ATM) , or other similar device generates data 
associated with a financial transaction. For example, 

10 a POS device may generate data associated with the 

sale of an item, whereas an ATM may generate data 
associated with a cash withdrawal by a customer. Due 
to human error, intentional misconduct, or machine 
malfunction, there may be a desire to display or 

15 analyze events associated with these financial 

transactions . 

Existing surveillance systems provide some 
monitoring of financial transactions. For example, 
some surveillance systems capture data associated with 

20 financial transactions for later analysis and 

reporting. Other surveillance systems store video 
images on videotape for later visual analysis and 
reporting of the event. Still other systems associate 
or overlay financial transaction data with video 

25 stored on videotape. 

SUMMARY OF THE INVENTION 

In accordance with the present invention, the 
disadvantages and problems associated with 
30 surveillance systems have been substantially reduced 

or eliminated. In particular, the present invention 
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provides a video surveillance system and method that 
combines data, video, and optionally audio associated 
with a financial transaction in a digital file. A 
server retrieves the digital file from the client and 
provides a graphical interface to retrieve the data, 
video, and optionally audio from the digital file for 
presentation and analysis. In another embodiment, the 
present invention provides a video surveillance system 
and method that combines data, video, and optionally 
audio associated with a financial transaction and 
transmits this information in real-time to a server. 
A server displays the data in a data window which is 
overlaid on a video image corresponding to the data. 

In accordance with one embodiment of the present 
invention, a video surveillance system includes a 
client that generates data associated with a financial 
transaction. The client has a camera that generates 
video associated with the financial transaction. The 
client stores data and video in a digital file. A 
server is coupled to the client using a communications 
network and receives the digital file from the client 
and stores the digital file in a memory. The server 
has a graphical interface that retrieves data and 
video from the digital file stored in the memory for 
presentation. 

In accordance with another embodiment of the 
present invention, a video surveillance system 
includes a client that generates data associated with 
a financial transaction. The client has a camera that 
generates video associated with the financial 
transaction. The client transmits the data and video 
over a communication network. A server is coupled to 
the client using the communications network and 
receives the data and video from the client . The 
server displays the video and data in real-time. 
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Important technical advantages of the present 
invention include the storage and/or real-time 
transmission and viewing of data, video, and 
optionally audio associated with a financial 
transaction. In particular, a client, such as a 
point-of-sale (POS) device like a cash register or an 
automated teller machine (ATM) , generates data 
associated with and/or upon the occurrence of a 
financial transaction. The client also includes a 
camera that generates associated video and optionally 
a microphone that generates associated audio. The 
client may store data, video, and audio in a single 
multimedia digital file. In a particular embodiment, 
the client includes two modes of operation. In the 
first mode, the client includes only data associated 
with the financial transaction in the digital file. 
In the second mode associated with an exception 
condition of the financial transaction, the client 
includes data, video, and optionally audio in the 
digital file. The exception condition may be defined 
by information transmitted from the server. 

The storage of different information into a 
digital file provides several important technical 
advantages. The digital file may be formatted, 
compressed, and communicated using digital 
communications technology. The digital file may be 
scrambled, rearranged, encoded, or otherwise processed 
to prevent tampering or disassociation of data, video, 
and audio. Also, a digital file format allows more 
sophisticated database storage, retrieval, and 
reporting functions . 

Another important technical advantage of the 
present invention includes a server coupled to the 
client that retrieves the digital file. The server 
includes a graphical interface that allows a user of 
the server to display, analyze, and generate reports 
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on information contained in the digital file. in a 
particular embodiment, the server is coupled to a 
plurality of clients, and each client generates 
digital files for financial transactions occurring at 
the client. The server collects and stores these 
digital files in a database. The graphical interface 
accesses the database to allow selection, 
presentation, analysis, and reporting of the financial 
transactions represented by the digital files. In a 
particular embodiment, data for each financial 
transaction appears as an entry in a table of 
financial transactions. Highlighted entries may 
indicate the existence of video associated with the 
data. The graphical interface may also include a 
video window for viewing associated video and a 
search/report window to allow selection and analysis 
of financial transactions. 

In another embodiment, the client provides a 
further technical advantage by transmitting data, 
video, and audio across a communications network in 
real-time. Additionally, data can be transmitted from 
the client to the server upon initialization of a 
real-time connection. This data may represent daily 
sales total, transaction totals, number of items sold 
since last contact or some other information. in this 
embodiment, a server coupled to the client receives 
the transmitted data and video. The server includes 
a display that allows the data to be shown as a data 
window overlaid on the associated video. Multiple 
data windows as well as multiple video windows can be 
displayed. In a particular embodiment, the server is 
coupled to a plurality of clients and each client 
transmits data and video for financial transactions 
occurring at the client. The server displays these 
transactions in multiple windows. An operator can 
change the views in the windows or the windows can be 
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changed automatically, based on some preexisting 
criteria. Other technical advantages are readily 
apparent to one skilled in the art from the following 
figures, descriptions, and claims. 

BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the present 
invention, and for further features and advantages, 
reference is now made to the following written 
description taken in conjunction with the accompanying 
drawings, in which: 

FIGURE 1 illustrates a video surveillance system; 

FIGURE 2 illustrates a client in the video 
surveillance system; 

FIGURE 3 illustrates a server in the video 
surveillance system; 

FIGURE 4 illustrates a graphical interface at the 
server in the video surveillance system; 

FIGURE 5 illustrates the components of an 
exemplary digital file used in the video surveillance 
system; 

FIGURE 6 is a flowchart of a method of operation 
of the client in the video surveillance system; 

FIGURE 7 is a flowchart of a method of operation 
of the server in the video surveillance system; 

FIGURE 8 illustrates a video surveillance system 
capable of real-time transmission of video and data; 

FIGURE 9 illustrates a display which includes a 
video window and one or more data windows; 

FIGURE 10 illustrates a display divided into 
multiple video windows and data windows; 

FIGURE 11 is a flowchart of real-time data and 
video transmission; and 

FIGURE 12 is a flowchart for a method of updating 
data from a client. 
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DETAILED DESCRTPTTON OP THR INVKNTTOM 

FIGURE 1 illustrates a video surveillance system 
10 that includes clients 12 coupled to servers 20 
using a communications network 24, in operation, 
clients 12 generate digital files 14 that include 
data, video, and optionally audio associated with 
financial transactions. Clients 12 communicate 

digital files 14 to servers 20 using network 24. 
Servers 20 store digital files 14 received from 
clients 12 in databases 18, and provide remote 
monitoring, reporting, and analysis of financial 
transactions occurring at clients 12. 

Clients 12 may include or be associated with any 
electronic device that generates data on a financial 
15 transaction, such as a point-of-sale (POS) device like 

a cash register, automated teller machine (ATM) , or 
any other appropriate device that generates data on a 
financial transaction. Clients 12 may be located at 
one or more sites, associated with one or more 
business organizations, or otherwise arranged or 
grouped in any appropriate manner. For example, two 
or more clients 12 may be co-located at a site, 
operated by the same business organization, or 
otherwise associated as indicated by bracket 26. Each 
server 20 in system 10 receives digital files 14 
associated with financial transactions occurring at 
one or more designated or associated clients 12. 
' System 10 contemplates any association or arrangement 
of clients 12 and servers 20 to accomplish remote 
monitoring and analysis of financial transactions. 

Network 24 represents hardware and software used 
in any suitable communications network or computer 
network, such as a local area network (LAN), wide area 
network (WAN), public switched telephone network, 
35 integrated services digital network (ISDN), switched- 

5 6 telephone network, private branch exchange (PBX) , 
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the global computer network known as the Internet, or 
any other appropriate technology or technique that 
allows components of system 10 to communicate 
information. Although client 12 and server 20 are 
referred to in the nomenclature of a client/server 
environment, it should be understood that client 12 
and server 20 may be any type of computer operating in 
any suitable environment that communicates using 
network 24 . Each component in system 10 includes any 
suitable hardware and software components to interface 
with and communicate using network 24 . 

In a particular embodiment, network 24 supports 
one-way and two-way audio/video conferencing. 
Throughout this description, audio/video conferencing 
includes conferencing of audio alone, video alone, or 
both audio and video, together with any associated 
data. For example, network 24 may include components 
to implement an integrated services digital network 
(ISDN) communications facility that supports the ITU 
H.320 video conferencing standard. In this 

embodiment, each component of system 10 may include 
appropriate transceivers, coders /decoders (codecs) , 
interface cards, and other hardware and software to 
implement audio/video conferencing and underlying data 
transfer. 

An alarm monitoring station 28 is also coupled to 
network 24 and detects alarm conditions at clients 12. 
In response to this detection, station 28 establishes 
communication with the particular client 12 that 
generates the alarm condition. Station 28 may display 
in a direct, dedicated, real-time, or near real-time 
fashion data, video, and audio generated at the 
particular client 12 that generated the alarm 
condition. Station 28 may also perform one-way or 
two-way audio/video conferencing with the particular 
client 12. In a particular embodiment, station 28 
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alerts and dispatches police, fire, security, or other 
officials to client 12. 

In operation, clients 12 perform financial 
transactions and generate digital files 14 associated 
with the financial transactions for storage in 
databases 16. At appropriate times, server 20 
receives digital files 14 from clients 12, and stores 
these digital files 14 in database 18. m one 
embodiment, database 18 maintained at server 2 0 
includes digital files 14 collected from numerous 
clients 12. Server 20 includes a database management 
system and a graphical interface to display, select, 
analyze, and report on financial transactions 
occurring at clients 12 that correspond to digital 
files 14 maintained in database 18. 

FIGURE 2 illustrates client 12 in more detail. 
On-site input/output devices 50 include microphone 52, 
speaker 54, cameras 55, and display 58. A video 
switch 60, coupled to cameras 56 and display 58, 
selects video from one or more cameras 56. A video 
cassette recorder (VCR) 62 or other appropriate 
recording device is coupled to input/output devices 
50, and records video and audio information on 
videotape 64 . 

Input/output devices 50 are coupled to a 
converter 70, which passes video 72 and audio 74 in 
digital format to a controller 76. Controller 76 is 
coupled to and receives data 82 regarding a financial 
transaction from ATM 78, POS 80, or any other device 
that generates data 82 regarding a financial 
transaction. An alarm 83 is also coupled to 
controller 76, and represents a motion detector, 
clock, panic button, or other device that generates an 
alarm condition 85 at client 12. 

Controller 76 is coupled to database 16 which 
stores digital files 14 and exception condition 84. 
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Exception condition 84 comprises information that 
directs client 12 when to store video and optional 
audio for particular financial transactions. For 
example, exception condition 84 may represent one or 
more activities, such as keystrokes at ATM 78 or POS 
80, that when detected in data 82 triggers the capture 
of video 72 and/or audio 74 for the financial 
transaction. Exception condition 84 may be defined as 
a noise threshold in audio 74 or a pixel or picture 
variance or difference threshold in video 72 that, 
when exceeded, triggers the capture of video 72 and 
audio 74. Controller 76 is also coupled to codec 86, 
which in turn is coupled to network 24 using interface 
88 . 

Particular components of client 12 may operate on 
one or more computers, shown generally as computer 90. 
Computer 90 maintains and executes the instructions to 
implement converter 70, controller 76, codec 86, and 
interface 88, and includes any suitable combination of 
hardware and software to provide the described 
function or operation of these components. Database 
16 comprises one or more files, lists, or other 
arrangement of information stored in one or more 
components of random access memory (RAM) , read only 
memory (ROM) , magnetic computer disk, CD-ROM, other 
magnetic or optical storage media, or any other 
volatile or nonvolatile memory. 

Computer 90 includes an input device 92 such as 
a keypad, touch screen, mouse, or other device, that 
can accept information. Output device 94, such as a 
computer display or speaker, conveys information 
associated with the operation of client 12, including 
digital data, visual information, or audio 
information. Both input device 92 and output device 
94 may include fixed or removable storage media such 
as a magnetic computer disk, CD-ROM, or other suitable 
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media to both receive output from and provide input to 
client 12. Processor 96 and its associated memory 
execute instructions and manipulate information in 
accordance with the operation of client 12. 

In operation, input/output devices 50 may operate 
in an analog or mixed analog/digital environment. For 
example, cameras 56 may generate and display 58 may 
display video in a standard television format such as 
NTSC or other analog signal format. Similarly, 
microphone 52 may generate and speaker 54 may convey 
audio information in analog form. if appropriate, 
converter 70 converts analog signals used by one or 
more input/output devices 50 into digital signals for 
video 72 and audio 74 used by controller 76. In one 
embodiment, input/output devices 50 generate and 
receive digital data and the operation of converter 70 
is unnecessary. 

Upon the occurrence of a financial transaction, 
ATM 78 or POS 80 generates data 82 associated with the 
financial transaction. Controller 76 analyzes video 
72, audio 74, and/or data 82 to determine if it 
indicates, corresponds to, or is associated with 
exception condition 84 stored in database 16. in a 
first mode, controller 76 determines that video 72, 
audio 74, and/or data 82 are not associated with 
exception condition 84 and stores only data 82 
generated by ATM 78 or POS 80 as digital file 14 in 
database 16. in a second mode, controller 76 
determines that video 72, audio 74, and/or data 82 
generated by ATM 78 or POS 8 0 is associated with 
exception condition 84, which triggers the capture of 
video 72 and optionally audio 74. Therefore, in the 
second mode of operation, controller 76 includes data, 
video 72, and optionally audio 74 associated with the 
financial transaction in digital file 14 stored in 
database 16. 
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Contemporaneously with the storage of digital 
file 14 in database 16 or at an appropriate later 
time, controller 76 retrieves one or more digital 
files 14 from database 16 for transmission to server 
20 using codec 86, interface 88, and network 24. 
Client 12 may schedule delivery of digital files 14 to 
server 2 0 in any appropriate manner. For example, 
client 12 may communicate digital files 14 to server 
2 0 at off-peak hours, at the end of a shift, at 
specified intervals during a day, week, or month, or 
at any other appropriate time, depending on the 
particular requirements of the business organization 
operating client 12. In addition, client 12 may 
initiate communication of digital files 14 in response 
to a command received from server 2 0 over network 24 . 
Also, alarm condition 85 generated by alarm 83 may 
cause client 12 to immediately communicate digital 
file 14 associated with alarm condition 85. In this 
embodiment, client 12 may transmit alarm condition 85 
to station 28 and establish a direct, dedicated, real- 
time, or near real-time one-way or two-way audio/video 
conference with station 28. 

In combination with or separate from the 
generation and communication of digital files 14, 
client 12 also supports one-way and two-way 
audio/video conferencing using network 24. For one- 
way audio/video conferencing, converter 70 passes 
video 72 from cameras 56 and audio 74 from microphone 
52 to controller 76. Controller 76 and codec 86 place 
video 72 and audio 74 into an appropriate format such 
as the video conferencing standard described in ITU 
H.320. Controller 76 may also include any data 
generated at client 12 in the conferencing 
information. In a particular embodiment, one-way 
audio/video conferencing signals are multiplexed and 
compressed onto a single digital bit data stream and 
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transmitted to server 20 or station 28 using the ISDN 
communications standard supported by network 24. 

For two-way audio/video conferencing, the 
components of client 12 perform the same outgoing 
conferencing capability, but also receive audio/video 
conferencing signals from server 20 using network 24, 
interface 88, and codec 86. Controller 76 receives 
incoming signals from codec 86, separates the signals, 
and passes video 72 and audio 74 to converter 70. 
Converter 70 performs conversion, if appropriate, and 
presents incoming conferencing signals to speaker 54 
and display 58. Controller 76 may also extract data 
from the incoming conferencing signals. 

In a particular embodiment, real-time video 72 
and optionally audio 74 is sent along with 
corresponding data 82 generated by ATM 78 or POS 80. 
The term "real- time" means real-time, near real-time, 
or contemporaneous as possible but subject to 
limitations in communication systems that cause 
substantial time to elapse between the capturing of 
video 72 and data 82 and the display at server 20. In 
this embodiment, video 72 and optionally audio 74 is 
sent to controller 76 where it combines with data 82 
from ATM 78, POS 80 or any other device that generates 
data 82 regarding a financial transaction. 
Alternatively, instead of controller 76 , any other 
device can be used that can combine video 72 and data 
82. Data 82 from more than one ATM 78 or POS 80 can 
be transmitted. Additionally, multiple video windows 
can be transmitted, each one representing a different 
camera 56 feed. These types of transmissions can also 
occur at multiple clients 12. Video 72 is transmitted 
along with the corresponding data 82 over network 24 
via interface 88. Alternatively, data 82 can be 
stored in database 16 over a period of time. Upon 
establishment of network connection or some other 
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occurrence, data 82 is transferred from client 12 to 
server 20. Server 20 can then query either its own 
database 18 or database 16 at server 12. 

FIGURE 3 illustrates server 20 in more detail. 
Input /output devices 100 include camera 102, 
microphone 104, and speaker 106. Input/output devices 
100 are coupled to a codec 108, which in turn is 
coupled to network 24 using an interface 109. A 
controller 110 is coupled to codec 108, display 112, 
and input devices 113. Display 112 displays 

information contained in digital files 14 received 
from clients 12. In particular, display 112 presents 
a graphical interface 116 that allows a user of server 
20 to display, select, analyze, and report on 
financial transactions occurring at clients 12 that 
correspond to digital files 14 maintained in database 
18. Also included in database 18 is a data 

configuration 15 which allows data to be overlaid on 
display 112 in a variety of formats. Input devices 
113 may include a keyboard, mouse, other pointing 
device, or any other appropriate input device that 
allows the user to interact with graphical interface 
116 and direct the operation of server 20. 

Controller 110 is also coupled to database 18, 
which stores digital files 14 received from clients 
12. Alternatively, a video cassette recorder 107 can 
be used to store real-time video 72. Database 18 
includes a database management system 114 that 
provides traditional database features to store, 
retrieve, and manipulate information stored as digital 
files 14 for monitoring, analyzing, and reporting on 
financial transactions occurring at clients 12. 
Database management system 114 supports any suitable 
flat file, hierarchical, relational, object-oriented, 
or parallel database operation. 
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In operation, server 20 receives data 82 , video 
72, and optionally audio 74 in the form of digital 
files 14 from network 24 using interface 109. Codec 
108 decompresses and converts this information into a 
proper format for storage in database 18 by controller 
110. The retrieval of digital files 14 from clients 
12 may occur on a periodic basis defined by clients 
12, a periodic basis defined by server 20, or as a 
result of server 20 polling clients 12 with commands 
to download information. 

In response to a request from controller 110, 
database management system 114 accesses selected 
digital files 14 and passes this information to 
controller 110 for presentation by graphical interface 
116 on display 112. Using graphical interface 116, 
the user can display, select, view, analyze, and 
report on information associated with financial 
transactions occurring at clients 12. 

In an alternative embodiment, server 20 receives 
data 82 and video 72 from network 24 in real-time. 
Video 72 is displayed on display 112 based on data 
configuration 15 stored in database 18. Overlaid as 
a data window on display 112 is a representation of 
data 82, such as a cash register receipt. 
Alternatively, multiple data windows can be displayed 
corresponding to data 82 from different ATM 78 or POS 
80 at the same or different location. Multiple video 
windows, each one a different video 72 from a 
different camera 56 at the same or different location 
can be shown on display 112. Data windows can be 
displayed for each video 72. A user can switch video 
72 based on what is occurring in a data window or 
views can be switched automatically based on some 
preexisting criteria. FIGURES 9-11 describe the 
techniques to display video 72 and data 82 in a 
variety of arrangements. 
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Additionally, upon connection between client 12 
and server 20, client 12 can automatically transfer 
data 82 to server 20. This can be all the financial 
records since last connection, all the financial 
records over a certain period of time, or some other 
configuration. Server 20 can then query either its 
own database 18 or the client ! s database 16 for 
further information. Data configuration 15 can 
control the display of data 82. For example, data 
configuration 15 can have the display show total sales 
for a certain time period broken down by categories of 
items purchased. FIGURE 12 describes in more detail 
techniques for updating data from a client. 

FIGURE 4 illustrates in more detail the 
components of graphical interface 116. Graphical 
interface 116 includes a table 120 having a number of 
entries 122 associated with financial transactions. 
Each entry 122 includes all or a portion of data 82 
generated by POS 78 or ATM 80 at client 12. 
Highlighted entries 124 may be emphasized by shading, 
font changes, color differences, or other appropriate 
technique to indicate the existence of associated 
video and/or audio. In a particular embodiment, 
entries 122 correspond to data received from clients 
12 operating in a first mode in which digital file 14 
includes data 82, and highlighted entries 124 
correspond to information retrieved from clients 12 
operating in a second mode in which digital file 14 
includes data 82, video 72, and optionally audio 74. 
In this manner, table 120 provides data 82 on 
associated financial transactions, and also conveys 
visually those financial transactions associated with 
particular defined exception conditions 84 at clients 
12. Highlighted entries 124 may then be quickly 
recognized by the user of server 20 and analyzed as a 
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suspect or more closely monitored financial 
transaction. 

Upon a user selecting highlighted entry 124 using 
input device 113, video window 126 presents a still 
frame or a selected portion of partial or full motion 
video 72 associated with the selected highlighted 
entry 124. Optional audio 74 may also be presented 
simultaneously with video 72. The user may manipulate 
a toolbar 128 to play, pause, stop, fast forward, 
rewind, adjust the volume, or perform other 
appropriate functions to analyze video 72 and audio 74 
presented by graphical interface 116. Furthermore, 
video window 126 may present a magnification box 130 
that allows the user to analyze selected portions of 
video 72 in more detail using zoom, pan, and other 
functions. The storage of vide 72 and audio 74 as 
digital information enables more sophisticated 
analysis techniques, such as the techniques provided 
by toolbar 128 and magnification box 130. 

Graphical interface 116 also includes a 
search/report window 132 that allows the user of 
server 20 to specify particular financial transactions 
to view in table 120. For example, search/report 
window 13 2 may prompt the user for a number of 
parameters that specify the desired financial 
transactions to view. Parameters may include time, 
date, store identifier, register identifier, amount of 
transaction, all transactions involving a particular 
item, all transactions meeting an exception condition, 
or any other appropriate parameter. Search/report 
window 132 may also include various printing, 
reporting, and analyzing capabilities of server 20. 

FIGURE 5 illustrates digital file 14 generated by 
client 12, optionally stored in database 16 at client 
12, and stored in database 18 at server 20. Digital 
file 14 includes data 82 generated by ATM 78 or POS 8 0 



at client 12, video 72, and audio 74. As described 
above, client 12 operating in a first mode in which 
exception condition 84 is not met may only include 
data 82 in digital file 14 . However, in a second mode 
in which exception condition 84 is met, client 12 may 
include data 82, video 72, and optionally audio 74 in 
digital file 14. 

Data 82 includes a transaction identifier 200, 
date and time 202, POS or ATM identifier 204, and site 
identifier 206, which may all be considered together 
as identifiers 208 that uniquely specify digital file 
14 in system 10. Data 82 also includes transaction 
data 210, that may specify transaction type, item 
identification, item cost, taxable amount, amount 
tendered, tax added, total, withdrawal amount, account 
information, user information, keys depressed at 
ATM 78 or POS 80 during the financial transaction, or 
any other data associated with the financial 
transaction. Controller 76 at client 12 may analyze 
transaction data 210 to determine if exception 
condition 84 is met. Data 82 may also include other 
data 212, such as a measure of the time the cash 
register door is open, an identifier of the employee 
on duty, an estimate of the number of persons in the 
store, or other information not directly related to 
the financial transaction but provided in data 82 for 
further analysis of the financial transaction. For 
clarity, digital file 14 in FIGURE 5 arranges 
information in blocks. However, data 82, video 72, 
and audio 74 may be arranged in any format or order, 
depending upon the particular implementation and 
technology used in system 10. 

The maintenance of data 82, video 72, and audio 
74 in a single digital file 14 provides several 
technical advantages. Digital file 14 may be 

scrambled, rearranged, encoded, or otherwise processed 
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to prevent tampering or disassociation of data 82 with 
its corresponding video 72 and audio 74. In addition, 
system 10 capitalizes on digital storage, compression, 
and communications techniques to quickly and 
efficiently gather digital information at server 20. 
Also, the digital format of digital file 14 enables 
more sophisticated database storage, retrieval, and 
reporting functions to be performed at server 20. 

FIGURE 6 illustrates a flow chart of a method of 
operation of client 12. The method begins at step 300 
where the next financial transaction occurs at ATM 78 
or POS 80. ATM 78 or POS 80 generate data 82 
associated with and upon the occurrence of the 
financial transaction at step 302. Continuously, upon 
the occurrence of the financial transaction at step 
3 00, or at any appropriate time, microphone 52 and 
cameras 56 generate audio/video information associated 
with financial transaction at step 304. Throughout 
this description, the term audio/video refers to video 
alone, audio alone, or both video and audio. If 
appropriate, converter 70 may generate video 72 and 
audio 74 for communication to controller 76. 

If controller 76 determines that exception 
condition 84 stored in database 16 is not met at step 
306, client 12 enters a first mode and stores digital 
file 14 including data 82 in database 16 at step 308. 
If exception condition. 84 is met at step 306, client 
12 enters a second mode and stores digital file 14 
including data 82, video 72, and optionally audio 74 
in database 16 at step 310. 

If controller 76 detects alarm condition 85 from 
alarm 83 at step 312, client 12 establishes 
communication with server 20 or optionally alarm 
monitoring station 28 at step 314. While client 12 
maintains alarm condition 85, client 12 and server 20 
or station 2 8 exchange data, video, and audio at step 
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316 to implement a one-way or two-way audio/video 
conferencing link for remote surveillance, management, 
or supervision. If alarm condition 85 persists at 
step 318, client 12 and server 20 or station 28 
5 continue to exchange data 82, video 72, and audio 74 

at step 316. If alarm condition 85 is over at step 
318 and the operation of client 12 is not done at step 
320, the method returns to process the next financial 
transaction at step 300. 

10 If no alarm condition 85 exists at step 312, 

client 12 determines whether to send digital files 14 
to its associated server 20 at step 322. If digital 
files 14 are to be sent to server 30, client 12 
selects digital files 14 to send at step 324. This 

15 may be done in response to a command received from 

server 20 or by specifying locally by client 12 or 
remotely by server 20 various parameters, such as 
time, site identifier, register identifier, or other 
appropriate parameter to select digital files 14. 

20 Controller 76 passes selected digital files 14 to 

codec 86 for formatting at step 326. Interface 88 
using network 24 sends digital files 14 to server 20 
at step 328. If appropriate, client 12 updates 
database 16 at step 330, for example, by deleting 

25 digital files 14 transmitted to server 20. If the 

operation of client 12 is not done at 320 or client 12 
determines not to send digital files 14 to server 20 
at step 322, the method returns to process the next 
financial transaction at step 300. 

30 FIGURE 7 illustrates a flow chart of a method of 

operation of server 20. The method begins at step 4 00 
where server 20 receives digital files 14 from 
associated clients 12. Server 20 stores the received 
digital files 14 in database 18 at step 402. Dashed 

35 feedback arrow 404 indicates that steps 400 and 402 

may execute in parallel, in series, in the background, 
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or in any other appropriate manner to receive digital 
files 14 from clients 12. 

A user of server 20 may input information into 
search/report window 132 of graphical interface 116 to 
select digital files 14 stored in database 18 to view 
at step 4 06. In response to the user's request, 
database management system 114 retrieves selected 
digital files 14 from database 18 and passes this 
information to controller 110 at step 408. Graphical 
interface 116 presents data 82 associated with 
retrieved digital files 14 as entries 122 and 124 in 
table 120 at step 410. Graphical interface 116 also 
highlights particular entries 124 with associated 
video 72 and optional audio 74 at step 412. 

Upon selecting particular digital files 14 from 
database 18 and presenting entries 122 and 124 in 
table 120, graphical interface 116 of server 20 
supports several user functions as illustrated by 
branch 414. Graphical interface 116 may support these 
functions in parallel, in serial, or in any other 
fashion to allow interaction with the user of server 
20. if the user selects highlighted entry 124 at step 
416, graphical interface 116 presents associated video 
72 and audio 74 in video window 126 at step 418. 
Graphical interface 116 then services functions of 
toolbox 128 and magnification box 130 at step 420 to 
allow further analysis of video 72 and audio 74 
associated with the selected highlighted entry 124. 

If the user selects an analysis or reporting 
function at step 422, graphical interface 116 services 
the analysis and reporting function at step 424 . For 
example, the user of server 2 0 may request summary 
statistics, print information, run predefined reports, 
or perform any other function on data 82 displayed in 
table 120 or maintained as digital files 14 in 
database 18. Graphical interface 116 outputs the 



results of the analysis and reporting functions at 
step 426 . 

The user may also request a new table at step 42 8 
by submitting another query in search/report window 
132. In response, the method continues at step 406 
where server 20 selects digital files 14 to view. If 
the operation of server 20 is not done at step 430, 
server ,20 continues to receive digital files 14 at 
step 400 and store digital files 14 in database 18 at 
step 402. 

FIGURE 8 illustrates a video surveillance system 
10 capable of real-time transmission of video 72 and 
data 82. Clients 12 are coupled to servers 20 via a 
communication network 24 as previously discussed in 
conjunction with FIGURE 1\ Each server 20 includes 
display 112 which is capable of displaying real-time 
video 72 and data 82 received from client 12. In this 
embodiment, video 72 is sent by client 12 over 
connection 24 as soon as it is captured by camera 56 . 
This is called real-time video, although it is 
understood that the limits of communication network 24 
and other components of surveillance system 10 may 
introduce an appreciable delay between the capturing 
of video 72 and its display at server 20. Data 82 
corresponding to video 72 is also sent along network 
24 . Additional data 82 from other sources at that 
location can be sent along network 24 as can 
additional video 72 from other cameras. . 

FIGURE 9 illustrates display 112 which includes 
video window 126 and one or more data windows 127. 
Data window 12 7 represents an overlay of data 82 from 
ATM 78 or POS 80 or any other device capable of 
producing data 82 generated by a financial 
transaction. For example, data window 127 may 
represent a live or scrolling version of a cash 
register receipt corresponding to the image of a 
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customer purchasing goods in a store in video windows 
126. Multiple data windows 127 can be overlaid on a 
video window 126, each corresponding to an ATM 78 or 
POS 80 at the same or different location. An operator 
at server 20 can switch video 72 corresponding to a 
particular data window 127 or the switching can be 
done periodically or automatically, based on a variety 
of criteria such as an alarm condition or preprogramed 
response. These include switching to video 72 based 
on data 82 (e.g., the presence of data 82, amount of 
money spent, key depressions at ATM 78 or POS 80, 
etc.), type of item purchased, movement in a video 
window, sound levels, or any other suitable criteria. 
Server 20 may switch, arrange, or display video window 

126 and/or data window 12 7 in response to these 
criteria. 

FIGURE 10 illustrates a display 112 divided into 
multiple video windows 126. These video windows 126 
can be from cameras 56 at the same or different 
locations. Data windows 127 are overlaid in each 
video window 126. Multiple data windows 127 can be 
displayed in a given video window 126 as discussed in 
FIGURE 9. Multiple video windows 126 and data windows 

127 may be repositioned, resized, or otherwise 
manipulated or arranged in response to criteria 
discussed above with reference to FIGURE 9. 

FIGURE 11 is a flowchart of real-time video 72 
and data 82 transmission. In step 500, client 12 
generates data 82 from a financial transaction at ATM 
78 or POS 80 or other device. Client 12 also 
generates video 72 corresponding to data 82 at step 
502. Client 12 transmits video 72 and data 82 over 
network 24 to server 20 in step 504. Video 72 and 
data 82 are communicated in real-time and can be from 
multiple sources. In step 506, data is overlaid as a 
data window 127 with video 72 on display 112. Server 
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2 0 overlays a single or multiple data windows 12 7 on 
a single or multiple video window 126 in any 
appropriate configuration as discussed above with 
reference to FIGURE 9 and 10. Step 508 determines if 
a change to a display needs to be made based on a 
variety of criteria. Manual changes, such as those 
initiated by an operator, are covered in step 510, 
These would include an operator switching to a window 
based on a transaction appearing in a data window or 
an operator switching camera views as part of a normal 
scan pattern. In step 512, server 20 automatically 
configures windows based on changes in video 72. For 
example, sudden movement may trigger a video window to 
appear. Also, if a camera becomes obstructed, a 
change in a video window might be triggered. In step 
514, display 112 automatically changes due to the 
presence or content of data 82, such as the amount of 
purchase, the type of purchase, some alarm condition, 
keystrokes, or other criteria. 

FIGURE 12 is a flowchart for a method of updating 
data 82 from client 12. In step 520, a link between 
client 12 and server 20 is established. This step can 
involve the actual establishment of a link over 
network 24, the restoring of a paused link, 
initializing the update procedure over an already 
established link, or some other connection criteria or 
prearranged transfer time. In step 522, client 12 
communicates data to server 20. This data 82 can 
contain register or item totals for a given time 
period, a running total since last connection, the 
total number of certain items sold, raw inventory or 
transaction data, or any other numeric or alphanumeric 
data that conveys information on the activity at 
client 12. Data 82 is typically stored in a file at 
client 12 and information gathered over a period of 
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time is accumulated in that file. This file can be 
stored at server 20 upon receipt. 

In step 524 server 20 displays data 82 in a 
format determined by data configuration file 15 stored 
in database 18. The display format may specify grand 
totals, sales on items of interest, inventory, cash on 
hand, or some other information. Configuration file 
15 may be stored as an initialization file or some 
type of configuration file. Configuration file 15 is 
designed to be updated easily to allow displays to be 
designed efficiently. Server 20 maintains multiple 
configuration files 15 based on client 12, the 
identity of the operator at client 12, the identity of 
the operator at server 20, a time measure (e.g., time 
of day, day of the week, day of the month, quarter, 
etc.), a particular report format, or any other 
appropriate criteria . 

In step 526, the process determines if additional 
queries need to be made. Additional queries can be 
made to extract data not transferred or to display 
data in an alternative format. If so, step 528 
determines if the queries are to be made locally, that 
is, at the server. If so, server 20 performs queries 
on the server's database 18 in step 530. If not, 
server 20 performs queries on the client's database 16 
in step 532. 

Although the present invention has been described 
in several embodiments, a myriad of changes, 
variations, alterations, transf ormations , and 
modifications may be suggested to one skilled in the 
art, and it is intended that the present invention 
encompass such changes, variations, alterations, 
transformations, and modifications as fall within the 
spirit and scope of the appended claims. 
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WHAT IS CLAIMED IS; 

1. A video surveillance system, comprising: 

a client operable to generate data associated 
with a financial transaction, the client having a 
camera operable to generate video associated with the 
financial transaction, the client operable to store 
data and video in a digital file; and 

a server coupled to the client using a 
communications network, the server operable to receive 
the digital file from the client and to store the 
digital file in a memory, the server having a 
graphical interface operable to retrieve data and 
video from the digital file stored in the memory for 
presentation . 

2. The system of Claim 1, wherein the client 
comprises a point-of-sale device and the financial 
transaction comprises the sale of an item. 

3. The system of Claim 1, wherein the client 
comprises an automated teller machine and the 
financial transaction comprises a cash withdrawal. 

4. The system of Claim 1, wherein the client 
comprises a microphone operable to generate audio 
associated with the financial transaction, the client 
operable to store data, video, and audio in the 
digital file. 

5. The system of Claim 1, wherein the digital 
file comprises a single multimedia digital file. 
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6. The system of Claim 1, wherein: 

the graphical interface at the server presents 
data as an entry in a table having a plurality of 
entries associated with a plurality of financial 
transactions; and 

the graphical interface at the server presents 
video in response to a selection of the entry in the 
table . 

7. The system of Claim 1, wherein the graphical 
interface at the server presents data as an entry in 
a table having a plurality of entries associated with 
a plurality of financial transactions, at least one 
entry highlighted to indicate the existence of video 
associated with the entry. 

8. The system of Claim 1, wherein: 

the client comprises a microphone, a speaker, and 
a display; 

the server comprises a camera, a microphone, a 
speaker, and a display; and 

the client and the server are further operable to 
conduct two-way audio/video conferencing. 

9. The system of Claim 1, wherein the client 
has a first mode and a second mode of operation, the 
client in the first mode includes data in the digital 
file, the client in the second mode includes data and 
video in the digital file, the second mode associated 
with an exception condition of the financial 
transaction. 

10. The system of Claim 9, wherein the server 
transmits information to the client to define the 
exception condition . 
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11. A video surveillance method, comprising: 
generating data associated with a financial 

transaction ; 

generating video associated with the financial 
5 transaction; 

storing data and video in a digital file at a 
client ; 

receiving the digital file at a server using a 
communications network; 
10 storing the digital file at the server; and 

presenting data and video in the digital file 
using a graphical interface. 

12. The method of Claim 11, wherein the client 
15 comprises a point-of-sale device and the financial 

transaction comprises the sale of an item. 

13. The method of Claim 11, wherein the client 
comprises an automated teller machine and the 

20 financial transaction comprises a cash withdrawal. 

14. The method of Claim 11, further comprising 
the steps of : 

generating audio associated with the financial 
25 transaction; and 

storing the audio in the digital file at the 
client . 



30 



15. The method of Claim 11, wherein the digital 
file comprises a single multimedia digital file. 
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16. The method of Claim 11, wherein the step of 
presenting comprises : 

presenting data as an entry in a table having a 
plurality of entries associated with a plurality of 
financial transactions ; 

receiving a selection of the entry in the table; 

and 

presenting video associated with the data in 
response to the selection of the entry in the table. 

17. The method of Claim 11, wherein the step of 
presenting comprises: 

presenting data as an entry in a table having a 
plurality of entries associated with a plurality of 
financial transactions; 

highlighting the entry to indicate the existence 
of video associated with the data; and 

presenting video associated with the data in 
response to the selection of the highlighted entry. 

18. The method of Claim 11, wherein the step of 
storing data and video at the client comprises: 

storing data in a first mode; and 

storing data and video in a second mode 
associated with an exception condition of the 
financial transaction . 

19. The method of Claim 18, further comprising 
the step of communicating information from the server 
to the client to define the exception condition of the 
financial transaction . 
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20. The method of Claim 18, wherein: 

the client comprises a point-of-sale device and 

the financial transaction comprises the sale of an 

item; and 

the exception condition comprises the activation 
of one of a selected no sale and void keys on the 
point-of-sale device. 
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■21. A video surveillance system, comprising: 

a client operable to generate data associated 

with at least one financial transaction, the client 
5 having a camera operable to generate video associated 

with the financial transaction, the client operable to 

transmit the data and video using a communications 

network; and 

a server coupled to the client using the 
10 communications network, the server operable to receive 

the data and video from the client and to display the 

video and data in real-time. 

22. The system of Claim 21, wherein the client 
15 comprises a point-of-sale device and the financial 

transaction comprises the sale of an item. 

23. The system of Claim 21, wherein the client 
comprises an automated teller machine and the 

20 financial transaction comprises a cash withdrawal. 



24. The system of Claim 21, wherein the client 
comprises a microphone operable to generate audio 
associated with the financial transaction, the client 
25 operable to transmit data, video, and audio over the 

communications network . 



25. The system of Claim 21, wherein the server 
forms a data window from the data and a video window 
3 0 from the video and overlays the data window on the 

video window . 



WO 98/01838 



PCTOJS97/12000 



31 

26. The system of Claim 21, wherein the server 
presents data from a plurality of financial 
transactions as a plurality of data windows, presents 
video from a plurality of video sources as a plurality 

5 of video windows, and associates the data windows with 

the corresponding video windows. 

27. The system of Claim 26, wherein the server 
receives user input to specify one of the data windows 

10 to display the video window associated with the 

specified data window. 

28. The system of Claim 26, wherein the server 
associated with the financial transaction 

15 automatically switches the video window to the video 

associated with the data in response to the presence 
or content of data. 

29. The system of Claim 26, wherein the server 
2 0 displays the appropriate video window and data window 

upon changes in one of the plurality of video windows. 

30. The system of Claim 21, wherein the client 
stores accumulated data associated with the financial 

25 transaction and transmits the data when the client 

communicates with the server. 
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31. A video surveillance method, comprising: 
generating data associated with a financial 
transaction; 

generating video associated with the financial 
5 transaction; 

transmitting data and video in real-time from a 
client; 

receiving the data and video at a server using a 
communications network; and 
10 presenting data and video on a display at the 

server . 



32. The method of Claim 31, wherein the client 
comprises a point-of-sale device and the financial 
15 transaction comprises the sale of an item. 



33. The method of Claim 31, wherein the client 
comprises an automated teller machine and the 
financial transaction comprises a cash withdrawal. 

20 

34. The method of Claim 31, further comprising 
the steps of : 

generating audio associated with the financial 
transaction; and 
25 transmitting the audio to the server. 

35. The method of Claim 31, wherein the step of 
presenting comprises: 

presenting data in a data window as a 
30 representation of the financial transaction; 

presenting video in a video window; and 
overlaying the data window on the video window. 
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36. The method of Claim 31, wherein the step of 
presenting comprises: 

presenting data as a plurality of data windows 
associated with a plurality of financial transactions; 
5 presenting video as a plurality of video windows 

associated with a plurality of video sources; and 

associating the data window with the 
corresponding video window. 

10 37. The method of Claim 36, further comprising 

the step of updating the video window and the data 
window in response to the presence or content of the 
data in one of the plurality of data windows. 

15 38. The method of Claim 36, further comprising 

the step of updating the video window and the data 
window in response to a change in one of the plurality 
of video windows . 

20 39. The method of Claim 36, further comprising 

the steps of: 

receiving a user selection; and 

updating the video window and the data window in 
response to the selection. 

25 

40. The method of Claim 31, further comprising 
the steps of: 

storing accumulated financial data in a file at 

the client; 

30 transmitting the file from the client to the 

server upon connection of the client to the server. 

41. The method of Claim 30, wherein the digital 
file contains financial records accumulated since last 

3 5 connection. 
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42. A video surveillance system, comprising: 
a client operable to generate data associated 
with a financial transaction and to accumulate and 
store the data as a digital file, the client having a 
camera operable to generate video associated with the 
financial transaction, the client operable to transmit 
the data and video across a communications network; 
and 

a server coupled to the client using the 
communications network, the server operable to receive 
the digital file upon connection with the client, to 
receive the data and video from the client and to 
display the video and data in real-time. 

43. The system of Claim 42, wherein . the client 
comprises a point-of-sale device and the financial 
transaction comprises the sale of an item. 

44. The system of Claim 42, wherein the client 
comprises an automated teller machine and the 
financial transaction comprises a cash withdrawal. 

45. The system of Claim 42, wherein the client 
comprises a microphone operable to generate audio 
associated with the financial transaction, the client 
operable to transmit data, video, and audio over the 
communications network . 

46. The system of Claim 42, wherein the server 
forms a data window from the data and a video window 
from the video and overlays the data window on the 
video window. 



WO 98/01838 



PCT/US97/12000 



35 

47. The system of Claim 42, wherein the server 
presents data. from a plurality of financial 
transactions as a plurality of data windows, presents 
video from a plurality of video sources as a plurality 

5 of video windows, and associates the data windows with 

the corresponding video windows „ 

48. The system of Claim 47, wherein the server 
receives user input to specify one of the data windows 

10 to display the video window associated with the 

specified data window. 

49. The system of Claim 47, wherein the server 
associated with the financial transaction 

15 automatically switches the video window to the video 

associated with the data in response to the presence 
or content of data. 

50. The system of Claim 47, wherein the server 
20 displays the appropriate video window and data window 

upon changes in one of the plurality of video windows. 

51. The system of Claim 42, wherein the client 
stores accumulated data associated with the financial 

25 transaction and transmits the data when the client 

communicates with the server. 

52. The system of Claim 42, wherein the server 
displays the digital file based on a configuration 

30 file. 
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53. A video surveillance method, comprising: 
generating data associated with a financial 
transaction; 

generating video associated with the financial 
transaction; 

storing accumulated data as a digital file; 
transmitting the digital file upon connection of 
the client and the server; 

transmitting data and video in real-time from a 
client ; 

receiving the data and video at a server using a 
communications network; and 

presenting data and video" on a display at the 



server , 



54. The method of Claim 53, wherein the client 
comprises a point-of-sale device and the financial 
transaction comprises the sale of an item. 

55. The method of Claim 53, wherein the client 
comprises an automated teller machine and the 
financial transaction comprises a cash withdrawal. 

56. The method of Claim 53, further comprising 
the steps of : 

generating audio associated with the financial 
transaction; and 

transmitting the audio to the server. 

57. The method of Claim 53, wherein the step of 
presenting comprises : 

presenting data in a data window as a 
representation of the financial transaction; 
presenting video in a video window; and 
overlaying the data window on the video window. 
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58. The method of Claim 53, wherein the step of 
presenting comprises : 

presenting data as a plurality of data windows 
associated with a plurality of financial transactions 
on a display at the server; 

presenting video as a plurality of video windows 
associated with a plurality of video sources on a 
display at the server; and 

associating the data window with the 
corresponding video window. 

59. The method of Claim 58, further comprising 
the step of updating the video window and the data 
window in response to the presence or content of the 
data in one of the plurality of data windows. 

60. The method of Claim 58, further comprising 
the step of updating the video window and the data 
window in response to a change in one of the plurality 
of video windows . 

61. The method of Claim 58, further comprising 
the steps of : 

receiving a user selection; and 

updating the video window and the data window in 
response to the selection. 

62. The method of Claim 53, wherein the digital 
file contains financial records accumulated since last 
connection . 
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- 19. The apparatus as claimed in claim 1 wherein the objects clustering 
application layer generates at least one new or updated object from the 
difference image and an at least one existing object. 

20. The apparatus of claim 1 wherein the object tracking program further 
comprises a scene characterization application layer for describing the 
scene and for triggering an alarm, based on comparing a behavior 
pattern of the at least one existing object to the at least one pre-defined 
behavior pattern or characteristic. 

21. The apparatus as claimed in claim 20 wherein the ' scene 
characterization application layer comprises an object movement 
measurement module for analyzing changes in the parameters of the at 
least one existing object and determining the at least one existing 
object movement 

22. The apparatus as claimed in claim 20 wherein the scene 
characterization application layer comprises an object merger module 
for correcting errors the at least one existing object and an alarm 
triggering mechanism for determining whether an alarm is to be 
triggered based on the at least one existing object patterns. 

23. The apparatus claimed in claim 1 wherein the background update 
application layer comprises a background draft updater module for 
updating the at least one reference image from the currently captured 
video frame. 

24. The apparatus claimed in claim 23 wherein the background update 
application layer further comprises a short term reference image, 
updater module and a long term reference image updater module for 
maintaining the updated short term and long term reference images. 

25. The apparatus claimed in claim 1 further comprising an object 
tracking control database, the database comprising; 

at least one long term reference image, the at least one long term 
reference image comprising a background image of the scene without 
dynamic or static objects tracked by the apparatus; 
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a short term reference image, the at least one short term reference 
image comprising a background image of the scene with the dynamic 
or static objects tracked by the apparatus. 

26. The apparatus claimed in claim 25 wherein the object tracking control 
database further comprising; 

an objects table comprising a list of the dynamic or static objects 
tracked by the apparatus, each object is associated with object data and 
object meta data; and 

a distance short term map and a distance long term map showing the 
short-term and long-term reference images; and 

a background draft comprising a changing image of the scene and 
making up the reference image. 

27. The apparatus claimed in claim 26 wherein the object tracking control 
database further comprising a discarded objects archive for storing 
discarded objects. 

28. A method for the analysis of a sequence of captured images showing 
a scene for detecting and tracking of at least one moving or static 
object and for matching the patterns of the at least one object behavior 
in the captured images to object behavior in predetermined scenarios, 
the method comprising the step of: 

capturing at least one image of the scene; 

pre-processing the captured at least one image and generating a short 
term difference image and a long term difference image; 

clustering the at least one moving or static object in the short term 
difference and long term difference images and generating at least one 
new object and at least one existing object. 

29. The method as claimed in claim 28 further comprising the steps of 
characterizing the visual scene and updating the background reference 
image by updating the short term reference frame and the long term 
reference frame. 
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30. The method as claimed in claim 28 further comprising the step of 
configuring the object tracking* program for providing at least one 
reference image, at least one timing parameter and at least one visual 
parameter. 

5 31. The method as claimed in claim 28 further comprising the step of 

configuring the object tracking program for setting at least one region 
of interest. 

32. The method as claimed in claim 28 further comprising the step of 
configuring the object tracking program, said step comprises the steps 

10 • of: 

constructing an initial short term reference image and an initial long 
term reference image; 

providing the object tracking program with the initial short term 
reference image and the initial long term reference image; 
15 providing timing parameters; and assigning visual parameters. 

33. The method as claimed in claim 32 wherein the step of constructing 
comprises creating the short term reference image and the long term 
reference image from a captured image. 

34. The method as claimed in claim 32 wherein the step of constructing 
20 comprises creating the short term reference image and the long term 

reference image from internally stored images. 

35. The method as claimed in claim 32 wherein the step of constructing 
comprises creating the short term reference image and the long term 
reference image through a learning process utilizing a set of 

25 sequentially ordered and captured images. 

36. The method as claimed in claim 28 wherein the step pf pre-processing 
comprises the steps of: 

obtaining the short term reference image; 
obtaining the long term reference image; ■ 
30 obtaining a currently captured image; 
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generating a short term difference image from the short term 
reference frame and the currently captured image; 

generating a long term difference image from the long term 
reference frame and the currently captured image. 
5 37. The method as claimed in claim 28 wherein the step of clustering 

comprises the steps of: 

building groups of clustered objects from at least two dynamic or 
static objects in accordance with the relative locations of each of the at 
least two dynamic or static objects; 
10 adjusting the parameters of each of the at least two dynamic or static 

objects clustered within each group; 

updating the parameters and status of each of the at least two 
dynamic or static objects. 

38. The method of claim 28 wherein the step of clustering comprises the 
15 steps of predicting the motion of the at least one moving object by 

predictive filtering and adapting the parameters of the at least one 
moving object. 

39. The method as claimed in claim 37 wherein the step of building 
groups of cloistered objects comprises the steps of: 

20 measuring the distance between each of the at least two dynamic or 

static objects; 

determining neighborhood relations between each of the at least two 
dynamic or static objects and in accordance with the results of the 
distance measurement; 
25 clustering the at least two dynamic or static objects in accordance 

with the determined neighborhood relations into distinct object, 
groups; and 

adjusting the distinct object groups in order to determine the optimal 
spatial parameters of each of the at least two dynamic or static objects 
30 in the distinct object groups. 
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40. The method as claimed in claim 38 wherein the step of adapting the 
. parameters of the at least one moving object comprises, the steps of 

locating the center of the at least one moving object; locating the 
" boundary points constituting the boundary line of the at least one 
5 moving object; re-calculating the location of the center of the at least 

one moving object; and inserting the at least one moving object into an 
objects table. 

41. The method as claimed in claim 40 further comprising the steps of 
adjusting the spatial parameters of the at least one moving object and 

10 retrieving similar objects to the at least one moving object from a 

discarded object archive. 

42. The method as claimed in claim 29 wherein the step of characterizing 
comprises the steps of: measuring the movement of the at least one 
moving object to determine the behavior of the at least one moving 

15 object; . merging spatially overlapping objects; generate an alarm 

trigger in accordance with the results of the behavior of the at least 
one moving object or in accordance with the spatial or visual 
parameters of the at least one moving object. 

43. The method as claimed in claim 42 wherein the alarm trigger is 
20 generated in accordance with the texture of the object. 

44. The method as claimed in claim 42 wherein the alarm trigger is 
generated in accordance with the shape of the object, 

45. The method as claimed in claim 42 wherein the alarm trigger is 
generated in accordance with the velocity of the at least one moving 

25 object. 

46. The method as claimed in claim 42 wherein the alarm trigger is 
generated in accordance with the trajectory of the at least one moving 
object. 

47. The method as claimed in claim 28 wherein the step of updating the 
30 background comprises the steps of: updating the background draft; 
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updating the short term reference image; and updating the long term 
reference image. . 
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